[Libbeat][New Processor] XML Decode #23678

P1llus · 2021-01-26T10:30:31Z

What does this PR do?

This PR adds 2 components to Libbeat.

It adds a small XML Unmarshal helper function to the common folder in Libbeat, the reason this is added is so that not only processors can utilize it if/when needed, but also inputs. For example I plan to reuse this in http_endpoint to add XML support there as well. Other beats/inputs that would benefit from this is Winlogbeat.
The helper function expects a []byte with valid XML object(s), and will return it as a struct reusing the names of the XML object(s).
It supports lists, arrays, nested fields, object identifiers (for example <book seq="1">).
If the XML is not valid it will return a proper error message describing why it failed.
For more information on supported formats, please see the included unit test files.
Second component is a xmldecode processor, this is the first processor I made, so please review it carefully. I have used urldecode, mime_type and the decode_json_fields processor as a guideline for this one.
For the supported objects and more information on supported formats, please see the included unit test files.

Why is it important?

This adds the possibility to read XML files for file/log input, or decoding XML strings in existing message fields, for example XML strings in existing JSON fields.
Implements an easy helper function for other parts of beats to reuse if they want to Unmarshal XML into a struct.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes [Discussion] Adding support/helpers/processors for XML in libbeat #23366

elasticmachine · 2021-01-26T10:30:33Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2021-01-26T10:30:33Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-01-26T10:45:04Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #23678 updated
Start Time: 2021-02-15T10:04:43.755+0000
Duration: 51 min 22 sec
Commit: 76f6582

Test stats 🧪

Test	Results
Failed	0
Passed	45880
Skipped	4762
Total	50642

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	45880
Skipped	4762
Total	50642

jsoriano

Thanks! This can be a nice addition. I have added some questions and suggestions.

libbeat/common/xml_test.go

libbeat/processors/xmldecode/docs/xmldecode.asciidoc

libbeat/processors/xmldecode/xmldecode.go

libbeat/common/xml.go

libbeat/processors/xmldecode/xmldecode.go

libbeat/processors/xmldecode/xmldecode_test.go

jsoriano

One minor thing, for the rest it LGTM.

libbeat/processors/decode_xml_fields/decode_xml_fields.go

elasticmachine · 2021-01-27T10:21:17Z

Pinging @elastic/agent (Team:Agent)

P1llus · 2021-01-29T09:46:13Z

Started off another build, as some CI's are failing, but did not seem to be because of my changes.

P1llus · 2021-01-29T16:10:03Z

jenkins run tests please

P1llus · 2021-02-03T15:24:58Z

jenkins run tests

jsoriano · 2021-02-04T09:52:36Z

/package

P1llus · 2021-02-04T18:05:03Z

@jsoriano do you have any idea how we can make these builds pass? Iwe rerun the builds about 6 times this week, last week the kibana snapshot made the builds fail and this week I am still a bit unsure.

andrewkroh

Looks like a nice addition. I'd like to take a look at it again after you address the comments.

libbeat/processors/decode_xml_fields/decode_xml_fields.go

libbeat/processors/decode_xml_fields/decode_xml_fields_test.go

libbeat/processors/decode_xml_fields/decode_xml_fields.go

libbeat/processors/decode_xml_fields/docs/decode_xml_fields.asciidoc

libbeat/processors/decode_xml_fields/decode_xml_fields.go

libbeat/common/xml.go

libbeat/processors/decode_xml_fields/decode_xml_fields.go

P1llus · 2021-02-08T12:28:10Z

Adding in the last changes, though still a few more left before a second review is ready.

P1llus · 2021-02-08T18:52:38Z

@andrewkroh
Added in the changes you requested, I also moved Target from single string to pointer again, as there was no easily or feasible way to determine if the value was null or empty string (for adding messages to root).

The libbeat helper was also moved to /libbeat/common/enc/mxj after talking to @urso

CHANGELOG.next.asciidoc

P1llus · 2021-02-09T11:05:35Z

jenkins run tests please

…entiate between null and empty string

…ng options

$ benchcmp old.txt new.txt benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15691 15686 -0.03% BenchmarkProcessor_Run/nested_and_array_object-12 39673 39098 -1.45% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 158 +0.00% BenchmarkProcessor_Run/nested_and_array_object-12 376 374 -0.53% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 8597 +0.00% BenchmarkProcessor_Run/nested_and_array_object-12 20310 19798 -2.52%

benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15686 8051 -48.67% BenchmarkProcessor_Run/nested_and_array_object-12 39098 20540 -47.47% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 75 -52.53% BenchmarkProcessor_Run/nested_and_array_object-12 374 184 -50.80% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 3520 -59.06% BenchmarkProcessor_Run/nested_and_array_object-12 19798 7824 -60.48% benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15686 8051 -48.67% BenchmarkProcessor_Run/nested_and_array_object-12 39098 20540 -47.47% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 75 -52.53% BenchmarkProcessor_Run/nested_and_array_object-12 374 184 -50.80% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 3520 -59.06% BenchmarkProcessor_Run/nested_and_array_object-12 19798 7824 -60.48%

P1llus · 2021-02-15T07:52:03Z

jenkins run tests please

kuisathaverat · 2021-02-15T08:46:23Z

lint stage failed

[2021-02-15T08:08:06.836Z] + make -C libbeat check

[2021-02-15T08:08:06.836Z] make: Entering directory '/var/lib/jenkins/workspace/Beats_beats_PR-23678/src/github.com/elastic/beats/libbeat'

[2021-02-15T08:08:13.413Z] >> fmt - go-licenser: Checking for missing headers

[2021-02-15T08:08:15.951Z] >> fmt - go-licenser: Adding missing headers

[2021-02-15T08:08:15.951Z] >> fmt - autopep8: Formatting Python code

[2021-02-15T08:08:15.951Z] >> fmt - goimports: Formatting Go code

[2021-02-15T08:08:42.504Z] >> check: Checking source code for common problems

[2021-02-15T08:09:21.252Z] # github.com/elastic/beats/v7/libbeat/common/encoding/xml

[2021-02-15T08:09:21.252Z] common/encoding/xml/decode_test.go:302:1: ExampleXMLToJSON refers to unknown identifier: XMLToJSON

[2021-02-15T08:09:26.523Z] Error: failed running go vet, please fix the issues reported: running "go vet ./..." failed with exit code 2

[2021-02-15T08:09:26.524Z] scripts/Makefile:152: recipe for target 'check' failed

[2021-02-15T08:09:26.524Z] make: *** [check] Error 1

[2021-02-15T08:09:26.524Z] make: Leaving directory '/var/lib/jenkins/workspace/Beats_beats_PR-23678/src/github.com/elastic/beats/libbeat'

P1llus · 2021-02-15T08:53:10Z

Thanks @kuisathaverat . Sorry for the build spam, it has had quite some issues with failing test runs the whole week, mostly unrelated however this one is related for sure.

P1llus · 2021-02-15T09:59:22Z

The name of the Example test for the XML package was not set according to what go vet expects, changed the name.

* stashing before initial commit * Initial commit * updating go.sum * updating it again * adding feedback from PR comments and removing expandkeys config entry * Updating changelog * removing expanded_keys from allowed fields * adding new changes based on PR comments, a few more changes remains * moving the xml decoder to its own subpackage based on PR comments * reverting back to Target being a string pointer, to be able to differentiate between null and empty string * Updating certain tests to fit the new ignore_failure and ignore_missing options * Updating unit test to test with missing field * updating license headers * adding benchmark test * benchmark, now also with allocation results * updating changelog entry * removing duplicate Changelog entry * changing changelog entry name to new name * Simplify error handling and fix race $ benchcmp old.txt new.txt benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15691 15686 -0.03% BenchmarkProcessor_Run/nested_and_array_object-12 39673 39098 -1.45% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 158 +0.00% BenchmarkProcessor_Run/nested_and_array_object-12 376 374 -0.53% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 8597 +0.00% BenchmarkProcessor_Run/nested_and_array_object-12 20310 19798 -2.52% * internal xml to json implementation * Use internal xml to json decoder benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15686 8051 -48.67% BenchmarkProcessor_Run/nested_and_array_object-12 39098 20540 -47.47% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 75 -52.53% BenchmarkProcessor_Run/nested_and_array_object-12 374 184 -50.80% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 3520 -59.06% BenchmarkProcessor_Run/nested_and_array_object-12 19798 7824 -60.48% benchmark old ns/op new ns/op delta BenchmarkProcessor_Run/single_object-12 15686 8051 -48.67% BenchmarkProcessor_Run/nested_and_array_object-12 39098 20540 -47.47% benchmark old allocs new allocs delta BenchmarkProcessor_Run/single_object-12 158 75 -52.53% BenchmarkProcessor_Run/nested_and_array_object-12 374 184 -50.80% benchmark old bytes new bytes delta BenchmarkProcessor_Run/single_object-12 8597 3520 -59.06% BenchmarkProcessor_Run/nested_and_array_object-12 19798 7824 -60.48% * changelog fix * Update docs * Add godoc example of xml to json * updating test name to fit Example naming convention Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 6839307)

…-arm * upstream/master: [Metricbeat][Kubernetes] Extend state_node with more conditions (elastic#23905) [CI] googleStorageUploadExt step (elastic#24048) Check fields are documented for aws metricsets (elastic#23887) Update go-concert to 0.1.0 (elastic#23770) [Libbeat][New Processor] XML Decode (elastic#23678) Fix: bad substitution of API key (elastic#24036) [Filebeat] Add Pensando DFW Module (elastic#21063) [Filebeat] Check if processor is supported by ES version (elastic#23763) Syslog system tests: be more forgiving (elastic#24021)

P1llus added enhancement libbeat Team:Integrations Label for the Integrations team Team:Security-External Integrations labels Jan 26, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 26, 2021

jsoriano reviewed Jan 26, 2021

View reviewed changes

libbeat/processors/xmldecode/xmldecode_test.go Outdated Show resolved Hide resolved

jsoriano approved these changes Jan 26, 2021

View reviewed changes

libbeat/processors/decode_xml_fields/decode_xml_fields.go Outdated Show resolved Hide resolved

jsoriano added the needs_backport PR is waiting to be backported to other branches. label Jan 26, 2021

andrewkroh self-requested a review January 26, 2021 21:26

andresrc added the Team:Elastic-Agent Label for the Agent team label Jan 27, 2021

andrewkroh requested changes Feb 4, 2021

View reviewed changes

urso reviewed Feb 4, 2021

View reviewed changes

libbeat/common/xml.go Outdated Show resolved Hide resolved

urso reviewed Feb 4, 2021

View reviewed changes

libbeat/common/xml.go Outdated Show resolved Hide resolved

urso reviewed Feb 4, 2021

View reviewed changes

libbeat/processors/decode_xml_fields/decode_xml_fields.go Outdated Show resolved Hide resolved

P1llus requested a review from andrewkroh February 8, 2021 18:51

urso reviewed Feb 8, 2021

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

P1llus and others added 12 commits February 14, 2021 15:34

reverting back to Target being a string pointer, to be able to differ…

7c24a56

…entiate between null and empty string

Updating certain tests to fit the new ignore_failure and ignore_missi…

99f7ad0

…ng options

Updating unit test to test with missing field

a816681

updating license headers

7e05b48

adding benchmark test

d61d3d5

benchmark, now also with allocation results

8085f42

updating changelog entry

52d9936

removing duplicate Changelog entry

fab3f86

changing changelog entry name to new name

3078147

internal xml to json implementation

03e4e3c

andrewkroh force-pushed the libbeat_xml_processor branch from f002a26 to d64e261 Compare February 14, 2021 20:36

andrewkroh added 3 commits February 14, 2021 15:38

changelog fix

bf9c310

Update docs

e5567e3

Add godoc example of xml to json

67ee664

andrewkroh approved these changes Feb 14, 2021

View reviewed changes

updating test name to fit Example naming convention

76f6582

P1llus merged commit 6839307 into elastic:master Feb 15, 2021

P1llus mentioned this pull request Feb 15, 2021

Cherry-pick #23678 to 7.x: [Libbeat][New Processor] XML Decode #24049

Merged

6 tasks

P1llus added v7.12.0 and removed needs_backport PR is waiting to be backported to other branches. labels Feb 15, 2021

marc-gr mentioned this pull request Mar 2, 2021

Create decode_xml_wineventlog processor #23910

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Libbeat][New Processor] XML Decode #23678

[Libbeat][New Processor] XML Decode #23678

P1llus commented Jan 26, 2021 •

edited

Loading

elasticmachine commented Jan 26, 2021

elasticmachine commented Jan 26, 2021

elasticmachine commented Jan 26, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

jsoriano left a comment

jsoriano left a comment

elasticmachine commented Jan 27, 2021

P1llus commented Jan 29, 2021

P1llus commented Jan 29, 2021

P1llus commented Feb 3, 2021

jsoriano commented Feb 4, 2021

P1llus commented Feb 4, 2021

andrewkroh left a comment

P1llus commented Feb 8, 2021

P1llus commented Feb 8, 2021 •

edited

Loading

P1llus commented Feb 9, 2021

P1llus commented Feb 15, 2021

kuisathaverat commented Feb 15, 2021

P1llus commented Feb 15, 2021

P1llus commented Feb 15, 2021 •

edited

Loading

[Libbeat][New Processor] XML Decode #23678

[Libbeat][New Processor] XML Decode #23678

Conversation

P1llus commented Jan 26, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Jan 26, 2021

elasticmachine commented Jan 26, 2021

elasticmachine commented Jan 26, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

elasticmachine commented Jan 27, 2021

P1llus commented Jan 29, 2021

P1llus commented Jan 29, 2021

P1llus commented Feb 3, 2021

jsoriano commented Feb 4, 2021

P1llus commented Feb 4, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

P1llus commented Feb 8, 2021

P1llus commented Feb 8, 2021 • edited Loading

P1llus commented Feb 9, 2021

P1llus commented Feb 15, 2021

kuisathaverat commented Feb 15, 2021

P1llus commented Feb 15, 2021

P1llus commented Feb 15, 2021 • edited Loading

P1llus commented Jan 26, 2021 •

edited

Loading

elasticmachine commented Jan 26, 2021 •

edited by jenkins-beats-ci bot

Loading

P1llus commented Feb 8, 2021 •

edited

Loading

P1llus commented Feb 15, 2021 •

edited

Loading